1. Field of the Invention
This invention relates generally to a method, system, and device for a chromakey function used in motion picture processing or composition. Such a chromakey function can provide a clean key signal and clean clipped-out foreground pictures without background colors. More specifically, the present invention pertains to a method, system and device for automatic determination of parameters which are used in a chromakey function as well as implementation of such method based on hardware means and software means. The present invention is related to technologies referenced in the List of References section of the present specification, and in brackets throughout the present specification, the entire disclosures of all of which are hereby incorporated by reference herein.
2. Discussion of the Background
Chromakey is a process used for special effects in motion picture and video industries where objects of interest against a backing color are shot into an electronic image, then this shot or image is used to generate a clipping signal called “Alpha”, “Key” or “Matte” in different application fields. It ranges from zero to unit and shows regions of the foreground objects, the background colors, and the transition boundaries between the objects and the background color. With such a clipping signal, a new picture can replace the backing color as a new background. The entire replacement process is technically called composition. The history and applications about chromakey and composition are well known and described in the prior art referenced to [1], [2], [3], [4], [5], [6] and [7], and in the List of References.
Solutions for a chromakey function in the prior art focuses on two issues:
Issue 1: pre-defines two-dimensional (2D) areas or three-dimensional (3D) volumes in a color space for background colors and foreground objects. Between foreground and background areas in the color space are transition color areas contributed by foreground objects' boundaries, semi-transparent objects, and shadows. Usually, the two geometrical shapes are not overlapped.
Issue 2: determines the parameters for the two geometrical shapes, such as the centroid of convex closed curves or volumes, sizes of shapes, alpha values between surfaces of two volumes, and so on. In this invention, the terms nominal backing color or reference background color refer to the centroid of a geometrical shape for the background colors.
Many patents and the literature can be found for Issue 1 on how to make better geometric shapes to classify and identify the color space, referenced to Appendix. A recent trend shows that instead of using geometrical shapes, people started to use more complex mathematical models in statistics to classify and identify color clusters. These mathematical models include Bayesian inference or MAP (maximum a posteriori) and PCA (principal component analysis), referenced to [8], [9], [10], [11], [12]. Although we see these methods published in the academic field and applied for still pictures or non-linear edition, we believe that these complex models will be applied to the motion picture industry with progress of semiconductor technology and computing devices in the future.
On the other hand, we barely find out the literature and patents on how to determine parameters for those geometric shapes, such as the nominal backing color and its range. Most of commercial products rely on user interface (UI) devices to implement Issue 2 as shown in FIG. 1. Through the UI devices, a single or multiple samples on the background is picked to determine the nominal backing color, which determines the centroid and range of the background area. And then multiple samples from foreground objects, and even the transition area are picked and used to tune the geometrical shapes in the color space until a satisfactory result is achieved. Usually, picking samples from a reference picture occurs in the first several frames of moving pictures. Such a manual determination process for chromakey parameters through UI devices has following disadvantages.
1) Sampling at different areas for the nominal backing color causes different results if background colors are not uniform. This is because the alpha value for each pixel is calculated in terms of this nominal backing color. If the nominal backing color changes, alpha values of pixels also changes. Usually, samples on the bright areas with high saturation give better results.
2) Once the nominal backing color is determined, it is hardly modified on the fly, because moving foreground objects on the fly could occupy any spot on background and do not give users enough time to pick proper background samples. When background colors change due to lighting, the predetermined nominal backing color certainly causes changes of alpha in the background area. For example, if a camera aims at a bright part of a large-area backdrop and takes samples for the nominal backing color but later it moves to a dark part of the backdrop, the entire background would become dark. This is because the dark background area appears like shadows in terms of bright reference color.
U.S. Pat. No. 5,774,191 discloses a method for determination of chromakey range based on histogram of one or more color component. It has following disadvantages.
1) Only one-dimensional histograms are used. Theoretically, a chromakey function is based on the 2D chroma vector or 3D color space, and hence color ranges must be determined by 2D or 3D histograms. However, 2D or 3D histograms require large memory devices to collect statistical data, which dramatically increases the cost of a chromakey function.
2) Due to one-dimensional histogram used, only rectangle shape or cubical shape is used to define the backing color region, which does not correctly separate foreground colors and background colors in many real cases.
U.S. Pat. No. 5,838,310 also discloses a method for determination of a key color range based on three histograms from color components R, G, and B respectively. Since this patent uses one-dimensional histogram, as U.S. Pat. No. 5,774,191 does, it has the same disadvantage as the above. Different from the prior art, U.S. Pat. No. 5,838,310 calculates histograms only on those pixels which are identified as the background colors. However, this method requires users to define background regions on a picture during initialization stage, and the optimum range for the key color depends on how to choose the background regions. Moreover, it requires an extra storage called plane memory to identify background pixels.